Human-Powered Top-k Lists

We propose an algorithm that obtains the top-k list of items out of a larger itemset, using human workers (e.g., through crowdsourcing) to perform comparisons among items. An example application is finding the best photographs in a large collection by asking humans to evaluate different photos. Our algorithm has to address several challenges: obtaining worker input has high latency; workers may disagree on their judgments for the same items; some workers may provide wrong input on purpose; and, there is a varying difficulty in comparing different items. We provide experimental evidence for the good performance of the algorithm, using extensive simulations and actual experiments with workers from Amazon’s Mechanical Turk.